Content Cooking
Overview Cooking content is how content is converted and massaged into the format that supports consoles (currently Xbox 360 and PS3). PCs, as well, can now use cooked data, which will result in much faster loading speed. For the most part it is pretty straightforward, but there are a few things at least one person on your team will need to know. The cooker itself is an Unreal commandlet. Generally, you will use the Unreal Frontend tool which will call out to the commandlet, passing in any configured options. However, you can also cook your content by directly using the commandlet, itself. Be sure you have set up your build! Cooking The cooker's primary role is to save packages in an optimal format for loading on a console (and optionally PC). The processing on packages may include the following: *Stripping out of any data that is not necessary such as editor-only data (this makes a large impact on file size) *Byte-swapping all data so that it can be loaded by big-endian architectures without CPU overhead for byteswapping *Processing resources so that they are stored in their native format and are ready for loading directly without any further processing on the console *Creating seek-free, self-contained packages for optimal loading from media with slow seek times (ie, DVD/BD) On consoles, UE3 only supports loading cooked packages. UE3 on PC can load cooked, uncooked, or even a mix of both. Determining what to cook The commandlet operates by loading packages, processing them, and then resaving them. The list of packages that are loaded is determined by the commandline and .ini settings. Cooked packages are saved to a cooked directory, with the format XXXGame\CookedYYY, where XXX is the name of your game and YYY is the platform name. The cooked package files have a .xxx extension on consoles, and the original extension on PC. By default only packages that have a newer date than the cooked package's date will be recooked, unless overridden by commandline options. One special file, which is Textures.tfc (.tfc for Texture File Cache), contains all the streaming textures. This is the default behavior for consoles, but can be disabled. Without the .tfc file, consoles will store a copy of referenced content packages with only their textures left in them. With the .tfc, the content packages are not needed, so the Cooked directory will only contain the seekfree packages and the .tfc file. In particular, the flow is this: *Non-seekfree packages (when NOT using .tfc): **Cook any out of date non-seekfree packages referenced by the seekfree packages being cooked *Seekfree packages (script, startup, maps) **Cook any seekfree packages that are out of date vs source files (checking if any referenced content packages are newer) **(When using .tfc) While processing seekfree packages, textures, or out of date textures, are added to the .tfc file for streaming *Standalone seekfree packages **Cook any standalone seekfree packages are out of date vs source file If -full or specified, then all cooked packages will be deleted first, so all files will fail the date checking and be cooked. If -recookseekfree, then maps, startup and standlone seekfree packages will be cooked no matter what. Commandline Generally, you will run the cooker with the Unreal Frontend, but if you want to automate cooking or similar, you will want to use the commandline interface. gamename.exe CookPackages map1 map2 ... mapn -platform= -options Options *platform= : Sets the platform to cook for. PLATFORM can be one of: **pc - cook PC with editor support **pcserver - cook PC for dedicated server (no editor support, lots of data stripped) **pcconsole - cook PC like console (no editor support, extraneous data stripped) **ps3, xbox360, xenon - console platforms **tegra, iphone - mobile platforms *full: Forces all of the existing packages to be resaved. If this option is specified, the contents of the CookedYYY folder is deleted first. *singlethread: Disables multithreaded cooking. *processes= : Uses N processes in a multithreaded cook. *recookseekfree: Forces all of seekfree packages to be recooked (startup, standalone seekfree, maps on commandline or .ini). Useful when the default dependency checking doesn't work (standalone seekfree packages in particular, because it won't recook all standalone seekfree packages when any non-seekfree packages change, unlike maps). *cookallmaps: Cooks all maps in the game. *mapsonly: Only cooks maps, will not attempt to cook anything else. *inisonly: Only cook the .ini and localization files (into the Coalesced.ini and Coalesced.int, etc files). *sha: Generate SHA hashes for the startup and coalesced files, and output to Hashes.sha. *languageforcooking= : Sets LANG as the language for cooking a single language. Defaults to int (English). *multilanguagecook= <+LANG2<-LANG3...>>>: Cooks multiple languages at the same time. Example: -multilanguagecook=int+ita-fra cooks English, Italian and French. Languages preceeded by a minus sign (French in the example) are only cooked for text localization so that only the startup packages and ini files will be cooked. *saveshadersatend: Tells the cooker to delay saving of shader cache until the end of cooking instead of after every package. *nopackagecompression: Disallows package compression of cooked packages. Speeds up cooking, at the expense of package size. *noloccooking: Only process the current language for subtitles and only generate the current language coalesced localization file. *verifytfc: Inspect the .tfc file for any errors *analyzereferecedcontent: Dump out stats of content referenced by cooked packages. See FAnalyzeReferencedContentStat. *usermode: Cooks content as if it's DLC or a mod (this will only cook what's specified on the commandline, it won't cook shipped script packages, etc). For more information about DLC, see DownloadableContent and DLCOnPS3. *skipmaps: Only cook non-map seekfree packages *skipsavingmaps: Cooks, but doesn't save, maps. Useful for cooking LOC data used by maps. *skipnotrequiredpackages: Skip loading & saving packages not required for cooking process to speed up LOC cooking. *skipmaterialcleanup: Skip cleaning up materials to speed up cooking iteration. *skipstaticmeshclean: Skip pushing materials onto the static mesh instances placed in the levels. skipmaterialcleanup will automatically do this, as well. *skippmapobjs: Skip the step of identifying objects contained in P-maps and removing them from sublevels it contains. (NOTE: Sublevels that are contained in multiple P-maps automatically skip this optimization.) *skippsysmodules: Skip the removal of duplicate modules from particle systems. *fastcook: The equivalent of specifying NOLOCCOOKING, SKIPMATERIALCLEANUP, SKIPPSYSMODULES, SKIPPAMPOBJS & QUICK. Output The ouput of the cooker is any of 6 types of files: levels, native script packages, a combined startup package, texture streaming packages, standalone seek-free files, texture file caches, and metadata/helper files. A level is a package that will contain all of the content it needs, except for high-level texture mips, making it basically seek-free to load a level (without the high detailed textures). A native script package is an UnrealScript package that has native classes in it and is always loaded by the game at startup. It will have content referenced by the script code cooked into it, so you need to be careful about how much content is directly referenced by script code. For Gears of War, we have one main native script package, WarfareGame.u, that has native base classes, and WarfareGameContent.u, which has no native classes, but has many subclasses that reference a lot of content that doesn't need to be loaded at all times (monsters, etc.). The combined startup package is a package that combines all packages needed for bringing up the game, but whose data isn't cooked into the main menu level. This includes objects like the DefaultMaterial. Any object in the startup package will be always loaded. Texture streaming packages contain the actual high-level (large) mip data, so that it is not duplicated too many times on the DVD, and can be streamed from at runtime easily. Standalone seekfree packages are packages that are not maps or script, but pull in all their references into a single package. This is used in Unreal Tournament for custom character piece packages. When constructing a character out of many pieces, we load the standalone seekfree package (asynchronously) to quickly load the meshes and their dependent textures. Because DynamicLoadObject is not supported on consoles, we use full package loading to load objects. When cooked, standalone seekfree packages can have 2 output packages - one with a _SF extension, and one without. The _SF file is the actual seekfree package that contains all objects and their dependencies, and the file without _SF contains the streaming textures (the engine does not stream textures out of seekfree files due to compression differences). A texture file cache is a single file that contains many packages worth of streaming textures combined into one file. This can be used to reduce the total number of files on disk and also the total number of open files at one time (in case a platform has any restrictions). Audio streaming packages are not created; however you can stream packages that have audio in them. This approach was favored instead of implementing support for simultanous streaming of arbitrary audio because we wanted to devote as much bandwidth to texture streaming as possible; and having a delay for such fast action games as Unreal Tournament and Gears of War would also be unacceptable, and syncing up the FaceFX anims becomes an additional chore at this point. As an alternative, a special dialog system was created for Gears. PC and Cooking The PC can run with cooked or uncooked data. To run the game with cooked data, use with the -seekfreeloading commandline option (or the Run with Cooked Data checkbox in Unreal Frontend). To run the editor with cooked data use the -cookededitor commandline option. Differences between cooking for PC and console: *PC packages keep their extension, console changes it to .xxx *PC non-seekfree packages retain all of the objects in them, not just textures like the console *Console cooking supports a TextureFileCache package *Console cooking removes more "raw data" than PC cooking, because the PC editor can load using cooked packages (see UT3) for mod support, etc. UT3 PC shipped with cooked data, and allowed for mods. We presented it to the user as "Publishing", which was similar to cooking, but it only cooked their packages in their mod. It is basically cooking in "-usermode" for the PC (ie, no cooking of our native script packages). Cooked Packages Types There are 4 types of cooked packages. Following are their descriptions and how they are loaded. Native script packages Description: These packages have native script code in them. These packages are cooked and made "seekfree" (although with script packages like Core and Engine, they can't really be 100% seekfree due to insane interdepencies between objects, which the bPreloadPackagesFromMemory option addresses by reading out of memory where seeking doesn't really matter). By cooking, all directly-referenced (using ' instead of " around their names) content (textures, meshes, etc.) are pulled into the package. Loading: They are 100% fully loaded all the time forever and ever :) They are loaded first thing at startup (ie Core, Engine, GameFramework, UTGame, etc). Startup packages Description: These packages are extra packages that are needed to initially boot the game, whose contents aren't cooked into other packages. They are listed in the Engine.StartupPackages section. The usual reason they are needed is that objects in these packages aren't referenced directly by script code or a map, so they need to be manually cooked and loaded. Loading: All objects in these packages are loaded at startup and will never be garbage collected. This means you should remove unnecessary objects from these packages to conserve runtime memory. Map packages Description: A map and all the content referenced by that map (except for classes in native script packages). Seekfree, just like native script packages. Multiple cooked maps may contain the same objects (two maps using the same texture will have the texture cooked into both). This increases DVD/BD usage, but not runtime memory, as the objects will be skipped over if they are already in memory from another map using them (in the case of streaming levels where you have multiple levels loaded). Loading: Loaded when game or player switches maps. It is fully loaded (often asynchronously for streaming etc) and the objects will be kept around until the level is unloaded from calling LoadMap or its streamed out. Texture packages Description: Packages that contain high-level mip data for texture streaming. During cooking, all but the high-level mips are removed from the package. Loading: These are never loaded as a package, they are only used for streaming textures (whose low-level mips are in a cooked package). Usage The method for cooking an asset depends on its usage: Directly referenced: If the object has a direct object reference to it, the seekfree packages (native script, maps) that needs it will pull it into the cooked seekfree package, and the asset package will not exist in the Cooked directory, unless it has streaming textures and you aren't using a texture file cache. Solution: Nothing to do. This is as expected. Loaded dynamically from any level: If no seekfree package (native script, map) references the object, and the object is loaded at runtime, we expect LoadPackage to be the way to load the object, since DynamicLoadObject will return NULL on a console unless the object is already loaded via LoadPackage. Solution: In Engine.PackagesToAlwaysCook section of your game's DefaultEngine.ini, mark it as a standalone seekfree package: Engine.PackagesToAlwaysCook +SeekFreePackage=MyGamePackage This will create MyGamePackage_SF.xxx (and MyGamePackage.xxx for any streaming textures if there are textures and you aren't using a TextureFileCache). This package will be in a more appropriate format for console loading than previously where it would not behave well with external references and streaming textures. Loaded dynamically from one level: If only one or two levels (i.e. menu level) dynamically loads the object, you can force the package into the map file, and then it will always be loaded while that map is loaded. Solution: In the Engine.PackagesToForceCookPerMap section of your game's DefaultEngine.ini, set the package to be forced into a map: Engine.PackagesToForceCookPerMap .Map=MyFrontend .Package=MyGamePackage Now, whenever MyFrontend is loaded, all objects in MyGamePackage will be loaded, and DynamicLoadObject=/=FindObject can be used to get it. Used in EVERY level: If the object can be used at anytime, you should put it into the Startup package, so that it will be loaded once ever, and always be in memory. Solution: In the Engine.StartupPackages section of your game's DefaultEngine.ini, add your package: Engine.StartupPackages +Package=MyGamePackage Configuration Configuration Files When setting your game up for cooking, there are some .ini settings you will need to modify. Since the console's Engine.ini file is generated from the main Engine.ini file, you should be able to specify these settings in your game's DefaultEngine.ini file. However, the cooker looks in the generated target platform's Engine.ini file for the settings, so you can specify them in your game's platform-specific Engine.ini file if you desire. Engine.PackagesToAlwaysCook Description: This section is for putting in maps that you want to make sure all users have cooked, as well as standalone seekfree packages. As of the Dec QA build, all non-seekfree packages will be cooked anyway, so there's no longer a need to specify random packages you want to be cooked. This section may be renamed and split into two sections in the future, as its use has changed so much over time. Also, standalone seekfree packages will not be recooked every time, unless overridden on the commandline. When to use: You would put packages that you must have around, like the main menu level. Also, when you need a standalone seekfree package (see above "Operation" section). Use "Package=" for maps to make sure are cooked, and "SeekFreePackage=" to specify seekfree packages. Format: Engine.PackagesToAlwaysCook +Package=entry +Package=UTFrontend +SeekFreePackage=CH_IronGuard_Male Engine.StartupPackages Description: The packages listed here are combined together into one seekfree startup package, per language (Startup_int.xxx, Startup_kor.xxx, etc). The objects in these packages (and their dependencies) are needed during initial boot of your game, and will always be in memory. Always. Also, objects in these packages will not be cooked into other packages. When to use: When you need objects during initial bootup or always in memory. In the example below, the appropriate font file will be used in place of UI_Fonts. So, if there was a UI_Fonts_KOR.upk, when cooking for Korean, it would put UI_Fonts_KOR objects into Startup_KOR.xxx. Format: Engine.StartupPackages +Package=FX_HitEffects +Package=UI_Fonts +Package=UI_Scenes_HUD +Package=UI_Skins +Package=UI_Skin_Derived +Package=SpeechRecognition bSerializeStartupPackagesFromMemory Description: Enables faster startup times at the expense of a significant amount of temporary memory during initial bootup. What this does at startup is to start reading in the list of startup packages into memory, asynchronously in the background. So, while some packages are being read off the (relatively slow) DVD, it can be processing other packages. By default, the engine will preload the native script packages, the startup map, and Startup_*.xxx. It works like this: *Kick off read of Core.u off DVD into an allocated chunk of memory *Wait for it to finish *Kick off read of Engine.u *Serialize (LoadPackage) Core.u out of the memory chunk into the engine *Continue reading packages while serializing already read in packages When to use: When you have enough memory that you are able to allocate these large additional chunks of memory, and you want to be able to boot up faster. Format: Engine.StartupPackages bSerializeStartupPackagesFromMemory=True bFullyCompressStartupPackages Description: Enables full-package compression of startup packages. These can be async decompressed in a single operation during load time. When to use: Combined with bSerializeStartupPackagesFromMemory, this will make for potentially faster background loads of the startup packages. By the time the engine gets around to processing the already loaded package, it will be already decompressed. Format: Engine.StartupPackages bFullyCompressStartupPackages=True Engine.PackagesToForceCookPerMap Description: This section will force certain packages to be entirely cooked into a map package. When to use: When you want to use DynamicLoadObject (DLO) to get load objects by string name instead of object reference. The cooker will only follow object referneces, not string references, so they won't cause objects to be cooked into the map. The UI system uses DLO to open UIScenes and textures, for instance. So, in Unreal Tournament, we force the packages that contain scenes and textures into the main menu. This will make it so that the scenes and textures will be loaded simply by loading the UTFrontEnd map. As long as the level is loaded, the objects will not be garbage collected. Format: The format is Map= followed by any number of Package= lines to list the packages that will go into the map. Engine.PackagesToForceCookPerMap .Map=UTFrontEnd .Package=UI_Scenes_Mutators .Package=UI_FrontEnd_Art .Package=UI_Portrait .Package=UI_Scenes_ChrisBLayout .Package=UI_Scenes_FrontEnd .Package=UI_Scenes_Common .Package=UI_Skin_Derived .Package=UI_Skins .Package=UI_Scenes_Campaign .Package=UI_Weapons .Map=UTM-MissionSelection .Package=UI_Scenes_Campaign .Package=UI_Skin_Derived .Package=UI_Skins Core.System SeekFreePCPaths Description: This section will retain the extension and relative directory structure inside the CookedPC directory (for editor functionality with -cookededitor flag). When to use: PC cooked builds. Format: Core.System SeekFreePCPaths= Source Content Cooking Category:UE3 Basics